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ABSTRACT 

Massive Open Online Courses (MOOCs) can be regarded as a promising next step in the evolution of distance 
education. However, they have been criticised for their poor learning design. This article describes the 
development of an adequate learning design in a series of nineteen MOOCs (called online master classes). A 
formative evaluation focuses on participation and user satisfaction. A total amount of 2083 individual learners 
enrolled in online master classes. Overall the user satisfaction is positive and stable. Thirteen pedagogical 
requirements for MOOCs form the output of this evaluation. It is concluded that the learning design that has 
been developed, matches with the pedagogical principles of distance education for adult learners. The format has 
proven to support more a diverse group of learners than the still dominant MOOC formats. 

INTRODUCTION 

Distance education allows learners and instructors to maintain communication although they are physically apart 
(Keegan, 1986). It has emerged from correspondence education, teaching by technical media (mass media radio 
and television) to digitized education (Peters, 2010). 

Massive Open Online Courses (MOOCs) can be regarded as a promising next step in the evolution of distance 
education. Gaebel (2013) describes MOOCs as online courses with no formal entry requirement, no participation 
limit, free of charge, but for which the learners do not earn credits. Originally MOOCs have their roots in the 
Open Education Movement that aims to enable education and knowledge development at no costs while the 
content can be reused and modified (Yuan & Powell, 2013). 

Despite of the potential, MOOCs have been criticised for their poor learning design (Cooper & Sahami, 2013; 
Gaebel, 2013; Yuan & Powell, 2013). Kalz and Specht (2013) have summarized that the current learning design 
of the two major design approaches is for the one format too open and unstructured and might therefore be only 
suited for a very self-directed population of learners with high media literacy (the so-called cMOOC). The other 
format is reproducing a classical lecture oriented approach without exploiting interaction and feedback 
opportunities (the so-called xMOOC). The Open University of the Netherlands (OU) has recently developed an 
approach to MOOCs that aims to improve the learning design. These MOOCs are offered to the learners as 
'online master classes'. They are aimed at adult learners whose primary objective is to keep up-to-date in their 
profession. 

The main characteristics of these online master classes are: 

• There is no limit to the number of learners. 

• Learners can participate at various levels of intensity. 

• It is an online format. 

• Every master class has a tum-around time (of a week) and learning goals. 

• They are freely accessible. Learners who want to receive a certificate pay a fee. They have access to 
additional resources. 

In this article we present the learning design that has been developed for these online master classes. In the next 
part we briefly discuss the state-of-the-art of the design and development of MOOCs in respect of pedagogical 
principles of distance education. Based on this discussion we describe the learning design of the online master 
classes. Furthermore we report about a formative evaluation. A list of 13 pedagogical requirements forms the 
result of this evaluation. In the last section conclusions are drawn and suggestions for future work are 
formulated. 

THE PEDAGOGICAL QUALITY OF MOOCS 

Although the first MOOCs were initiated in 2008, the interest in MOOCs has increased since Ivy League- 
universities like Stanford and Harvard started to offer online courses for free on a large scale (Sharpies et al, 
2013). According to Yuan and Powell (2013) MOOCs offer the opportunity for massive up-scaling of course 
participants and increasing access of education, for example to reach working professionals as a new target 
group (Gaebel, 2013). However, Yuan and Powell (2013) stress concerns about the pedagogy and quality of 
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current MOOCs, mainly due to their poor learning design. The question is if the learning design of these 
MOOCs matches with the pedagogical principles of distance education for adult learners. 

According to Holmberg (2003) distance education does not only imply non-contiguous teaching and learning, it 
also includes mediated interaction between learners and instructors (two-way traffic). Additionally, Holmberg 
(2003) stresses the importance of interaction for supporting learners, and personal relations including feelings of 
empathy and belonging (to motivate learners). Wagner (1994) defines interaction as “reciprocal events that 
require at least two objects and two actions. Interactions occur when these objects and events mutually influence 
one another”. Interaction has been regarded as important for processing information, for learner control, for the 
adaption of learning programs based on learners’ input, for constructing knowledge and for the creation of 
learning communities (Anderson, 2003). 

Rhode (2009) concludes that engagement in interaction can lead to enrichments of learning, while it can be 
identified as a central element and catalyst of engaging learning experiences. According to Khalil and Ebner 
(2013) the interaction quality and quantity in online learning environments are important factors that influence 
learner satisfaction and indirectly influence dropout rates. 

In education three types of interactions can be distinguished: student-content, student-instructor and student- 
student. Student-student interaction is the exchange of information and ideas amongst learners. Real-time 
presence of an instructor is not necessary to facilitate this. Student-instructor interaction is interaction between 
learner and expert to foster improved understanding of the content by learners, such as student guidance or 
feedback. Alexander (2003) emphasizes the importance to acknowledge that there is a difference between 
experts and professionals who are competent but not experts. Therefore these so-called ‘non-experts’ should be 
able to learn from experts. Moore (1989) described student-content interaction as "a defining characteristic of 
education”. Processing tasks like assignments and tests are examples of student-content interaction. This third 
type of interaction facilitates processing and internalization of information students encounter in learning. 
Student-content interaction results in improved understanding, changed perceptions and changes in cognitive 
structures of the mind of the learner (Moore, 1989). 

In his theory of transactional distance (TD), Moore (1997, 2007) claims that in distance education the essential 
distance is a pedagogical one between teacher and learner and not a time- and space-dependent distance. 
Psychological and communication distance creates the transactional distance, which may lead to insufficient 
reciprocal understanding. According to Moore (2007) this distance needs to be reduced to facilitate learning. The 
key components to minimize transactional distance are dialogue between learners and teachers to support 
learning, flexibility of structure of the educational program (e.g. learning goals and meeting individual learning 
needs) and autonomy of the learner over the learning process. If dialogue, flexibility of structure and learner’s 
autonomy are decreasing, the transactional distance will increase. 

Vaughan and Carlson (1992) and Jansen (2004) underline that structuring education improves study progress. 
Although this requirement is developed on basis of research on curricula, one could argue that structure supports 
learning in other learning activities as well. 

Hrastinski (2007) emphasized the different purposes of asynchronous and synchronous communication. He 
concludes that synchronous communication supports a more intense interaction that stimulates personal 
participation and motivation, while asynchronous communication facilitates cognitive and reflective types of 
participation. Both forms of communication complement each other. 

According to Merrill (2002) learning takes place when learners have to cope with real-world problems, when 
relevant previous experience is activated, when instruction demonstrates what has to be learned, when learners 
are required to apply new knowledge to solve problems, and when learners are encouraged to integrate new 
knowledge in their daily practice. This implies more learner control. 

Clark and Mayer (2011) discern three domains of learner control (content sequencing, pacing and access to 
learning support). According to Clark and Mayer (2011) there is evidence that dynamic adaptive control on 
learning by experienced learners leads to better learning results than program control by the provider. 
Additionally, they conclude that learners should have pacing control and that navigational support has to be 
offered. 

Roediger and Karpicke (2006) conclude that testing is a powerful way of improving memorization of learning 
content. In fact, tests enhance later retention more than additional study of the material. Moreover, feedback is 
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not a necessity to foster retentions. They conclude that self-tests, which can be done recurrently, can be used to 
improve retrieval of information. This phenomenon is known as the testing effect (Roediger & Karpicke 2006). 
This implies that tests not only can be used for assessment or explicit activation of prior knowledge, but also as a 
learning activity on its own. 

Currently from the perspective of theories on distance education, the learning design of the dominant format of 
xMOOCs can be compared with the e-leaming courses of the beginning of the 21st century: 

• Fixed design with repeated activities. 

• Focus on learner-content interaction and learner-learner interaction (not embedded in the curriculum of 
the course). 

• Assessments with multiple-choice questions. 

• Focus on instruction. 

• No adaption of the content of the course is possible during runtime (for example based on the feedback 
of learners). 

Thus our aim was to develop a new learning design for MOOCs, based on theories of distance education. A 
secondary goal was to attract a group of learners that did not participate in learning activities of the Open 
University of the Netherlands before. 

DEVELOPMENT PROCESS 

For the development of the online master classes we applied an iterative process that was oriented at principles 
of agile software development (see Beck et al. 2001). According to agile software development requirements can 
change during the development process in which ‘business people’ and developers collaborate together. Changes 
to the format are delivered frequently (e.g. in a timescale of a couple of weeks) while there is always a working 
solution in use. 

Three teams worked together during the reflective development process. A technical development team has been 
responsible for delivery platform, technical functionalities and the design of the template, used for online master 
classes. A pedagogical development team has been responsible for the development of a pedagogical approach. 
The third team has been responsible for efficient process for the organization and realization of online master 
classes. 

The first three online master classes were exploratory in nature: the pedagogical concept and the different 
technologies were explored and tested. After each run of an online master class we evaluated all its aspects: 
technically, content, format, experiences of learners, and the contribution of experts, moderator and session 
leader. Based on this qualitative assessment we proposed and implemented improvements. After the third online 
master class we used an online questionnaire for the learners to assess their opinions about the online master 
class. The results of these evaluations were used as one of the inputs for improvements of the design. 

THE LEARNING DESIGN OF THE ONLINE MASTER CLASSES 

The pedagogical development team developed the setup of the learning design of online master classes, based on 
distance education theory (described above). In doing so the pedagogical development team was confronted with 
an important design challenge: realizing a balance between flexibility and the three types of interaction. 
Flexibility and freedom of choice for learners on one hand, and coherent activities (like related instructions, 
interactions and processing tasks) for the benefit of learning on the other hand are correlated. Thus it is important 
to find the right balance for the target group of online master classes. 

The setup of the learning design of the online master classes is presented in figure 1. 
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Figure 1: Activity diagram of the learning design of online master classes 

In this figure the flow starts at ‘orientation’ and end with ‘archive’. In the first step learners orientate themselves 
on the content and procedure of the online master class. After enrolment, they are able to view additional 
information about the subscribed learners, and the preparatory learning activities. At the start of the online 
master class, the first additional learning activities are provided to the learners. They can access different 
learning activities each day. A variety of learning activities have been included into the learning design to 
stimulate active participation, like asynchronous and synchronous discussion and reflection. Learners are 
stimulated to demonstrate their knowledge in the group, apply the knowledge on challenging problems/cases, 
and are challenged to provide examples of the application of the knowledge in their practice. The first day is 
focused on orientation and activation of prior knowledge. An orientation test and self-test are used for these 
purposes. Moreover, the self-test is used for improving memorization of learning content. An asynchronous 
discussion in an available group-wall is used for orientation as well. During the second day learners are able to 
make assignments to deepen their theoretical insights on the topic of the online master class and to apply 
relevant theories in their daily practice. Learners study resources like articles and papers to conduct this 
assignment. An example of a theoretical assignment is: "Study the Horizon report on higher education. Choose 
one of the described six trends, and describe this trend using the presented theoretical framework of Sharpies 
(page 17-21). Please focus on the level of the learning activity". 

The third day is dedicated to the live session. The discussion assignments provide input for the live session. 
During a live session learners view a live stream video and have the possibility to use a chat. Learners make 
remarks or ask questions. Other learners can respond. A moderator (backstage) selects questions and submits 
these questions by a separate chat channel to the session leader. During the live session the session leader asks 
these questions to the expert, who responds. 

When the live session is over, the recording and the chat discussion are made available for learners. Moreover, 
learners have the opportunity to continue working on the assignments of the first two days. At day 6 of the week- 
long format the paper presentation of the PhD students takes place to stimulate theoretical depth. Again, live 
video streaming in combination with the chat box is used. The online master class is concluded with an online 
evaluation. 
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FORMATIVE EVALUATION 

The first three online master classes (October 2011-December 2011) were considered as a pilot and were not 
evaluated with a questionnaire. 

Between January 2012 and March 2013 nineteen online master classes were organized, with a diverse set of 
topics. Twelve topics were related to the field of technology-enhanced learning (such as ‘Tablet computers in 
education’ or ‘The future of the virtual learning environment’). The other seven subjects were related to learning 
sciences in general (e.g. ‘Effective learning strategies’ or ‘Education for highly gifted students’). 

This section provides an overview of the learners, the user satisfaction and a list of pedagogical requirements 
that are the result of the formative evaluation. 

Overview learners 

Table 1 provides an overview about the total number of learners, the mean number of learners, the ratio between 
OU-employees, OU-students and external learners and the ratio between male and female learners. The number 
of learners ranged from 91 learners to 448 learners. Seventy one per cent of the learners did not take part in any 
educational activity of the organization previously. The gender ratio was balanced. Since we did not know in 
advance which target group would be attracted by the educational format we have also analysed the age range of 
learners as depicted in figure 2. 


Table 1: Learner details online master classes 


Total number of learners 

Mean number of learners 
per online master class 

Ratio OU-employees/OU-students/ 
External learners 

Percentage 

Female/Male 

2083 

180 

OU-employees: 6% 
OU-students: 23% 

External registered learners: 71% 

Female: 54% 
Male: 44% 



Figure 2: Age learners per online master class 


In fourteen online master classes the largest group of learners was 45-55 years old. In some cases the largest 
group was older than 55 years. Hardly anyone younger than 25 years joined the online master classes. 89% of the 
learners were older than 35 years. 

The majority of learners only visited one online master class (63 %). 17% attended two online master classes, 
while 11% participated in three online master classes. The rest of the learners visited four or more online master 
classes (1,7% of the learners joined 16 or more online master classes). 

User satisfaction 
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In the evaluation of online master class 4-7 five questions were asked related to satisfaction (5-Point Likert 
scale). Table 2 presents an overview of these questions, including the scores (mean of the ratings of the four 
online master classes, range of standard deviation). 


Table 2: Satisfaction master classes 4-7 



Mean 

Range SD 

The online master class has motivated me from the beginning to the end 

3,01 

0,9337-1,0787 

The online master class has provided new insights 

2,88 

0,8771-1,1430 

The online master class inspired me to do things differently 

2,73 

0,9258-1,1260 

The online master class was useful time spent 

3.31 

0,9134-1,1364 

The online master class inspired me to further thought 

3,49 

0,8732-1,1162 


The learners were satisfied with three of four online master classes. One online master class received on every 
question a score lower than 3. A qualitative analysis of the open questions indicated that there were technical 
issues during the fifth online master class that might have influenced the score. 

Moreover, the following open question was asked: Did you find the content of the online master class 
sufficiently practical or did you prefer a more scientific approach to the subject? The answers showed that 
learners had different expectations about the online master classes. An analysis of the response to this question 
showed that in general, there was a group of learners that did prefer a more scientific approach, while another 
group wanted to learn about practical guidelines. A third group believed that there was a sufficient balance 
between theory and practice. 

In the evaluation of online master class 8-22 learners were asked to grade the online master class on an ordinal 
scale from 1-10. Table 3 presents the outcomes: 


Table 3: User satisfaction online master classes nr. 8-22. 


Online master class nr. 

Mean 

SD 

08 

6.87 

1,01 

09 

7,32 

1,03 

10 

6,85 

1,26 

11 

7,00 

0,66 

12 

6,65 

1,58 

13 

6,60 

1,23 

14 

7,10 

1,18 

15 

7,57 

0,53 

16 

7,50 

0,55 

17 

7,86 

0,69 

18 

6,87 

0,78 

19 

6,20 

1,09 

20 

5,75 

2,63 

21 

7,00 

1,15 

22 

7,50 

0,52 


Based on internal evaluations and user feedback it was concluded that online master classes should increase their 
scientific depth. Moreover, it was concluded that online master classes should have additional possibilities for 
activation of prior knowledge and enhancing retrieval of information. From the ninth online master class an 
online paper presentation by PHD-students was added. The intention of this online paper session was to provide 
learners, who were more interested in research and theory, the opportunity to gain more scientific insights. 
Furthermore, an orientation test was used to activate prior knowledge, while a self-test was used to improve 
retrieval of information. 
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Despite the changes in the learning design of the online master classes the satisfaction of learners stayed on a 
relatively high level. The learners positively reviewed all evaluated online master classes, except three. Five- 
point Likert scales were used to ask if learners expected to receive practical guidelines and to identify if learners 
expected to hear new theoretical views. The average scores of the first question ranged from 3,16 to 4,6 (SD: 
0,53-1,72), while the average scores of the second question ranged from 3,78 to 4,91 (SD: 0,35-1,30). Learners 
still expect to receive practical guidelines and scientific insights in an online master class. 

Pedagogical requirements 

The following pedagogical requirements can be regarded as the backbone of a learning design for online master 
classes for professional development. They are the result of the application of theory on distance education 
(described previously), and the evaluation of the organized online master classes. 


Table 4: Pedagogical requirements learning design online master classes 


Nr. 

Description requirement 

How applied 

1 

Online master classes should provide learners 
with knowledge sources and learning activities 
that keeps them up-to-date in their profession. 

The selection of subjects of online master classes is 
based on emerging educational research and trends. A 
variety of knowledge sources and learning activities are 
offered. 

2 

Online master classes should foster learner- 
control about the intensity of participation. 

Learners are able to spend as much as time in the 
learning activities as they want. Learners have the 
flexibility to participate in learning activities. Except the 
live sessions learners can learn when they want. 
Recordings of live sessions can be viewed anytime. 

3 

Online master classes should foster learner- 
control about selecting learning activities they 
want to do. 

Learners have the freedom to select in which learning 
activities they want to participate. 

4 

Online master classes should use the 
experience of learners as a starting point and 
input for learning. Learning activities and tools 
foster sharing experiences. 

Learners are able to share their experiences in 
asynchronous discussions (based on assignments) and in 
the chat during the live session. Learning task should 
foster application in the context of learners. 

5 

Online master classes should have a reciprocal 
design. Experts are able to follow the learner to 
interactions between learners. If needed -for 
example in case of misconceptions- the expert 
will intervene and support the learners. Based 
on these interactions, the content of the online 
master class will be adapted. 

Experts monitor interactions. They intervene to foster 
deep learning and to prevent misconceptions. The 
content of the live sessions is partly based on the 
asynchronous interactions preceded to the live session. 
After finishing the live session, the expert reviews and 
responds to questions that could not be answered during 
the live session. 

6 

Online master classes should foster the 
distinction between experts and non-experts. 
Non-experts should have the opportunity to 
interact with experts. 

Experts provide live sessions and other learning 
activities. During live sessions learners are able to 
interact with experts by a moderated chat. Furthermore, 
learners interact with experts during asynchronous 
discussions. 

7 

Online master classes should stimulate retrieval 
of information and activation of prior 
knowledge. 

An orientation test has been implemented to activate 
prior knowledge. A self-test has been implemented to 
stimulate retrieval of information. 

8 

Online master classes should foster learner- 
content interaction, learner-instructor 
interaction, and learner-learner interaction. 

Assignments, the orientation test, the self-test are used 
for learner-content interaction. Asynchronous 
discussions and the live session are used for learner- 
instructor interaction. Learner-learner interaction is 
facilitated in asynchronous discussions and during the 
live session. 

9 

Online master classes should stimulate active 
participation. 

Assignments are used to stimulate active participation. 

10 

Online master classes should stimulate learners 
to apply what they have learned, and to 
integrate the new knowledge in their daily 
practice. 

Assignments are used to foster application of new 
knowledge. 

11 

Online master classes should have synchronous 
sessions that foster the involvement and 

The format of online live sessions was an interview. An 
interviewer interviewed an expert. Questions by learners 
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motivation of the learners. 

in a moderated chat were used as input for the interview. 
A live session lasts one hour. The alternation between 
speaker, interviewer and learners fostered the 
involvement and motivation of the learners. 

12 

Online master classes should have an explicit 
structure of learning activities that are 
conducted during a defined amount of time. 

An explicit structure of learning activities, with a turn¬ 
around time of a week, had been applied to stimulate 
participation. Learners could decide when and how 
intense they wanted to leam, within the framework of 
this structure. 

13 

Online master classes should foster learners to 
increase scientific depth. 

Adding an online paper presentation by PhD-students to 
the learning design of the online master classes 
enhanced the balance between the theoretical 
background and practice-related knowledge. The 
intention of this online paper session was to provide 
learners, who were more interested in research and 
theory, the opportunity to gain more scientific insights. 
During a separate live session two PhD-students gave a 
fifteen minutes-presentation about a research paper on 
the subject of the online master class. At the end of the 
presentation the PhD-students answered several 
questions that were asked by learners in the chat. As a 
consequence student-student and student-instructor 
interaction were enhanced. 


DISCUSSION 

This article has described the development of an adequate learning design for MOOCs. Based on theory and 
experiences we formulated 13 pedagogical requirements that have guided the iterative development of the 
learning design in the various runs. This educational format did attract a group of learners that previously did not 
participate in learning activities of the Open University. Seventy one per cent of the learners in online master 
classes were no employees or current students of our university, and 89% of the learners were older than 35 
years. 

An important design challenge was to find a balance between flexibility and the three forms of interaction. Our 
assumption was that, the option to be able to choose the amount of time spent for the new format would 
contribute to satisfaction of learners. In general, this assumption can be confirmed. The explicit breakdown of 
the learning activities in a turn-around time of one week led to more structure. Nevertheless, the use of more 
structure restricts flexibility, while flexibility is seen as an important characteristic of a learning design for 
distance education. Moreover, more options for interaction might suggest an increased workload for learners. 
Although learners still are able to decide what they want to leam and how much time they want to invest, the 
changed structure and possibilities for interaction may indicate that the flexibility to leam 'just in time' and 'just 
enough' has declined. In addition, synchronous sessions limit the possibility to leam whenever learners want 
although they can view the recordings in their own time. 

With exception of three online master classes, the user satisfaction is positive and stable. The positive and stable 
user satisfaction can be interpreted as an indication for the learners' appreciation of the learning design, including 
the pedagogical requirements. Arguably, an explanation is that the balance between easy accessible participation 
and flexibility for learners and consistent learning activities (including interactions) has been realized. However, 
user satisfaction is influenced by more factors than the learning design. 

The educational format of online master classes complies with important pedagogical principles described in the 
theory. Arguably, this result combined with the enrolment of hundreds of learners and user satisfaction nourishes 
the conclusion that online master classes are an appropriate format for distance education for adult learners. 

Currently, the learning design of the dominant format of MOOCs (cMOOCs and xMOOCs) is not based on 
pedagogical principles of distance education for adult learners. Since online master classes have similar 
characteristics as a MOOC, application of the pedagogical requirements and the learning design of online master 
classes will improve the learning design of the dominant format of MOOCs and will enhanced the design 
towards the diverse group of learners that is participating in these open courses. For example live sessions, 
instead of pre-recorded videos, will enhance learner-instructor interaction. However, the question is if live 
sessions will foster learner-instructor interaction if thousands of learners participate in these sessions. Such a 
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large amount of concurrent users would be lead to an unmanageable tsunami of interactions. However, it is 
questionable if MOOCs are an effective format for distance education when meaningful learner-instructor 
interactions are impossible to have. While some of the experiences from designing the format of online master 
classes can inform the design of Massive Open Online Courses, some aspects of the design might have a more 
limited transferability. A turnaround time of a week was consciously chosen to offer professionals an 
opportunity to plan and manage their learning activities in a reasonable amount of time. In a classical higher 
education context these turnaround times could be much longer. 

CONCLUSIONS AND FUTURE WORK 

We have developed an adequate learning design for MOOCs that is attractive for new groups of learners. The 
online, accessible and flexible characteristics of the format, combined with expert knowledge and interaction 
possibilities, provide opportunities for large groups of professionals to keep up with their discipline while the 
pedagogical requirements and the learning design fit with the characteristics based on distance education 
theories. Different forms of interaction have been used in the online master classes: synchronous (chat), 
asynchronous (especially online forums and group-walls), among learners, with content and with 
instructors/experts. Research underlines the importance of these interactions from the perspective of learning. 
The developed learning design and pedagogical requirements will lead to stronger design for MOOCs, compared 
to the current designs. This format is less open and unstructured than the design of the cMOOC format that 
attracts self-directed learners with high media literacy. Furthermore, the developed learning design and 
pedagogical requirements will lead to a more learner-controlled and interactive design than the classical lecture 
oriented xMOOC format (Kalz & Specht, 2013). 

However, there are still plenty of questions left for future research. Further improvements and especially further 
evaluations of the learning design are needed. The user evaluation is now restricted towards satisfaction. This is 
informative, but not sufficient to access the real fit with the learners needs. Obviously, satisfaction is not an 
indicator of learning results or improvement of performance. These effects are not evaluated, while they could 
have an impact on the learning design of online master classes. 

Finally, it should be investigated more thoroughly if the learning design of the online master classes is applicable 
on large-scale courses with thousands of learners. The question is if there is a maximum to the amount of 
learners that can participate in a MOOC that fosters meaningful learner-instructor interaction. 
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